What features do you want in new format?

[spwolf]

Originally Posted by TBGBe

Not sure what you mean with these existing formats (conversion?)-
zip
cab
setup.exe/data1.cab
tar/tar.gz/tgz/tar.bz2
rar

re-compression... if you compress them as they are, you will gain nothing... if you extract and compress them, between 15 and 50% gain in compression is possible.

[Mameluke]

I would most likely want to see the following in this order if possible please

1. Internal and External (RAR style) recovery record (with choice of percentage just like RAR).
2. JPEG compression (got many pictures that are quite large)
3. Strong Encryption with ability to encrypt file names (Like Winrar and 7z)

Thank you



Mame

[davidsplash]

would love to see a format that actually can compress media files better than we have at present. At the moment most media formats are donme by several formats i would like one format to do them all e.g wave mp3 and jepg format compression all in one archive format at present i have to use several formats in order to get the best comopression for each media format like sound or images

[davidsplash]

a format that preserves data like who made it and what version of the archive format made it. usefull for people who need this info

[By-Tor]

I'd still like to see wz jpeg compression for zipx.

[spwolf]

Originally Posted by By-Tor

I'd still like to see wz jpeg compression for zipx.

it is still eventual possibility... however, while we researched jpeg compression last year, our first version was slightly better compressing than WZ Jpeg and 3x faster on dual core machine... and that could be done even better with 20% better compression than wz and up to 3x faster speed on dual core machines .

so which one would you want?

[By-Tor]

Originally Posted by spwolf

it is still eventual possibility... however, while we researched jpeg compression last year, our first version was slightly better compressing than WZ Jpeg and 3x faster on dual core machine... and that could be done even better with 20% better compression than wz and up to 3x faster speed on dual core machines .

so which one would you want?

Why not both?

[spwolf]

it takes 2 months for engineer to implement each, so during those 2 months he could do:PDF, MP3, PNG instead etc, etc.

it is very time consuming and demanding project, which is why nobody but PA can read these archives currently.

[By-Tor]

Ah. From the sound of it, I thought the implementations were close to complete. I guess the first thing I'd like to see then is the wz jpeg compression, just for compatibility. I'm less concerned with a unique format right now as portability issues would be a hindrance. For another unique archive format to catch on, I think you'd need to release an open source portable command line version at the very least before it would be truly useful among other users.

[spwolf]

Originally Posted by By-Tor

Ah. From the sound of it, I thought the implementations were close to complete. I guess the first thing I'd like to see then is the wz jpeg compression, just for compatibility. I'm less concerned with a unique format right now as portability issues would be a hindrance. For another unique archive format to catch on, I think you'd need to release an open source portable command line version at the very least before it would be truly useful among other users.

compatibility with only WinZip though, no other utility, and certainly no free or open source utility... so what is the difference really between PAF, IZEHRBLAH, and ZIPX, when only two products support extracting ZIPX archives.

Problem is that ZIP itself, due to old nature of format, will never be anywhere as good as modern format, and whatever extensions are added to ZIPX not only that they cant be as good as new format, they will also not be compatible with other products either.

Who knows, maybe PAF will be open format and maybe there will be free extraction tools, eventually anyway

[By-Tor]

WinZip has a large userbase, regardless of whether we like it or not. 7-zip can handle most of ZIPX, with the exception of jpeg compression, but, Igor has stated he would implement some form of it if he saw a need.

ZIP by its nature is just a container format. So, you're right, it doesn't matter much what you call it. I'm just thinking about trying to capture the largest audience possible.

I'm by no means anti-PAF. heh. I'd love to see a better format. After seeing so many archive formats come and go, I know it takes time for any format to catch on if at all.

[spwolf]

Originally Posted by By-Tor

WinZip has a large userbase, regardless of whether we like it or not. 7-zip can handle most of ZIPX, with the exception of jpeg compression, but, Igor has stated he would implement some form of it if he saw a need.

ZIP by its nature is just a container format. So, you're right, it doesn't matter much what you call it. I'm just thinking about trying to capture the largest audience possible.

I'm by no means anti-PAF. heh. I'd love to see a better format. After seeing so many archive formats come and go, I know it takes time for any format to catch on if at all.

yeah, but there are much bigger differences to be made with new, proper format... you cant just put something into zip and get good results, it doesnt work that way.

compress folder with zipx lzma and with 7zip lzma, and notice that the difference in size could be up to 50%, simply because zip doesnt and will not ever have proper solid compression.

point with paf is that we can make something unique with it that nobody else has - compression for many popular formats... weather it becomes next biggest thing is something else, we plan to use it in our next backup utility as well, where nobody does any of these things and space savings will be significant.

[spwolf]

but pretty much, everything on that list takes at least a month to implement for single engineer... thats why nobody has done it before (expensive) and why all the new formats are very much alike (variations of old).

[By-Tor]

Originally Posted by spwolf

point with paf is that we can make something unique with it that nobody else has - compression for many popular formats... weather it becomes next biggest thing is something else, we plan to use it in our next backup utility as well, where nobody does any of these things and space savings will be significant.

I totally agree. I wasn't clear on how far reaching you wanted things to be.

[spwolf]

Originally Posted by By-Tor

I totally agree. I wasn't clear on how far reaching you wanted things to be.

it is massive undertaking though, and it takes a while...

but it will be worth it for instance for backups, where we will be able to do 50% more efficient backups that rest of the current utilities

[spwolf]

keep in mind we have been "researching" this for past 2 years, so we are not staring from 0 here...

things we did so far:
- ver 1 jpeg compression - 3x speed of wz jpeg with slightly better compression. Aim in PAF: maintain similar super speed while adding extra 20% compression compared to WZ Jpeg.
- ver 1 differential and versioning system - compared to leading Mozy backup service, we had 30% smaller differential backups (which means 30% faster, 30% more effcient, 30% less costly for bandwith and storage space), without strong compression implemented... Goal is to tie it together into PAF and enable stronger compression that would give us 50% gain total.
- Various multicore research into improving speed for other operations during compression, not just compression codec, that should add extra 20-30% speed improvements over current formats (with similar codecs).
- modification of lzma2 codec into pa-lzma, to fit our format better, and better overall performance (to be released as open source), this is 80% done as it is...

[spwolf]

New formats idea site:
http://ideas.powerarchiver.com/

[Socrates]

The sample size on your survey is so small, I would caution against taking it too seriously.

If you found a way to ask a broader audience what they want, I would be shocked if multi-volume support would be in the top five.

There may not be a better way to reach a broader audience, and, if so, having established voting, you probably have to act based on the suggestions you received. Still, I fear that in so doing you be will be spending a lot of effort on a feature that really won't appeal to that many people. I suspect 5-7 of the other options would be more appealing, even if they are not ones I would use.

So if you try this again, you might explore ways of getting input from a larger sample.

[spwolf]

no need to worry, while ideas site is interesting we still have our own schedule and goals to work with (which is getting better compression on things that are not compressible currently).

multi volume feature is really simple to implement, but we probably would not do it that way if we didnt get enough votes.

[Socrates]

If it is simple, then that's a compelling reason to do it.

What I often don't know is how difficult it is to implement a new feature. Some might seem easily, but be difficult (or next to impossible) while others that seem difficult might be a snap.

[spwolf]

Originally Posted by Socrates

If it is simple, then that's a compelling reason to do it.

What I often don't know is how difficult it is to implement a new feature. Some might seem easily, but be difficult (or next to impossible) while others that seem difficult might be a snap.

hardest things on that lists are new codecs for pdf, jpeg, mp3... thats both hard and time consuming. Everything else on that list will probably take less time all together (!) then building special jpeg compressor.

[Socrates]

I suppose I would have thought special compression would be especially difficult. But not that much more difficult.

Thanks for letting us know.

At some point (and if not too complicated), could you explain the relative difficulties of building new compression from pds versus jpgs. I assume the former would be easier (more white space). Perhaps, though, that assumption is borne of ignorance.

[spwolf]

#1 goal for the format: special codecs for file types that are most used currently yet can not be compressed by current tools.

basically this is compression for complicated (already compressed) file types.

so what you have to do is take file apart, and divide it into parts that can and that can not be compressed (which is why jpeg, png, docx, pdf that is compressed, can not be compressed further usually).. then compress the part that can be compressed with special codec designed for that format. All of this is done transparently to the user of course, and usually quite fast if done right.

But for each format (mp3, jpeg, png, pdf, docx, odt, etc), special codec is required. So there is a lot of development work to be done here. There are also no examples of such work, and only very few utilities do it - for instance, Stuffit has a lot of special compressors, but it is available only if you give out your credit card (no actual free trial), while most other utilities do not have anything but general codecs.

Reality is that most people compress things that are already compressed, so using zip, or rar or 7zip on most things people usually backup or send via email will not result in great savings, or sometimes savings at all. If you compress jpegs to send over email or upload somewhere or simply backup, you will not gain any compression. On the other hand, with special jpeg codec, you can expect 20%-30% gain on your full album of pictures.

For instance zip has one most common codec which is deflate. WinRar has single codec too.
Now .paf/pa/power will have 5-6 at least within next 2 years. So you can imagine how big the task is. But the gains are big too so it is worth it.

[spwolf]

difference between pdf and jpeg is that while for both, you have to develop special recompression algorithm, pdf uses deflate compression from zip and then its contents once opened up, can be compressed well with general codecs.

on the other hand, jpeg has many variations so first recompression has to be able to take apart various jpegs and then you have to build completely custom codec to compress that picture inside.

So basically it is double the work compared to PDF.

Advantage with PDF is that we can use a lot of the code to recompress other formats like PNG, DOCX, ODT, SWF, since they all use deflate to give them (weak) compression.

For instance, DOCX containts XML files inside that are compressed with deflate. However since they are compressed, if you try to compress them again, you will gain very little... but if you unravel weak compression and then apply stronger one, big gains are possible.

Here is test example.

1. Contract in DOCX format - 104 KB
2. DOCX compressed with 7zip - 95KB
3. DOCX recompressed properly - 64 KB

So thats 40% gain on DOCX file for instance. Imagine if you have many of them on your computer, or if your company sends many via email or backup service... Time and cost savings are quite significant here.

Now actual % gained is different for different formats and there are further optimizations possible (for instance detect pictures and text inside single file differently and compress them with their own codecs), but you can see how much potential this has.

Main thing here is that it has to be done seamlessly and it has to be fast, otherwise people will not use it. And then we come to #2 part of new format - multicore optimizations.

[Socrates]

thanks!

[Sir Richard]

I would have to say that Jpeg, and PDF Compression is a good idea! as your possibly aware I have brought Picture compression up a few times in the past with a good warrent for it.

But with a list aslong as the above there are so many other good things to choose from.

I have made some votes. )

[davidsplash]

i would like the new format to introduce some sort of protection so that if the archive is changed it tells you by whom and can stop this if you dont wnat it updated but would like people to extract files from it only

[davidsplash]

One idea would be to have time limited archives that actually destroy themselves after a set amount of time. saw this in america well worth it if you are sending things over the internet. They also had archives that only let you open them a set number of times.

[davidsplash]

to sound a downer you can not have every feature in the new format as that would slow it down.

What we would like to see is several new archive formats .

For instance pdf compression

wave compresion

etc

would love to see all these new formats use the plugin feature in pa.

[FiatLux]

The option that you when in the "with options" interface then may choose both Encrypt and make SFX in one operation, and implemented using a technique that can handle both many andd/or large files....