Yeah I was talking to Krdax the other day about secret hush hush stuff, and he had the interesting claim that (at least on his system), animated textures were significantly increasing his ram hit. I think the thing to take away from that is that a good animated texture sheet needs to be, at minimum, around 1024x1024 if not larger. For some cluttered projects this may simply not be viable. I planned for this ahead of time.
I gut the base game's directory. Any meshes or textures I don't use that I can safely remove gets deleted. This should be standard practice for any mod riding the ram limit. It's easy to do for the end user, and can probably be automated in a .bat file. Alternatively, I think you could replace them with 0kb files in the mod, but I haven't tried it yet.
Retribution makes extensive use of unique textures (I have over 300 individual textures dedicated to particles alone) for particles. My textures are higher resolution, my ships are considerably higher poly, I use larger unit counts and have significantly more guns per ship than in Salvation. Yet I've managed to increase the project's performance to the point where I can run a four-way fleet fight on high (for the project) unit counts with fps significantly higher and more consistent than Salvation in a two-way fight on low unit counts. As anyone who saw the original project knows, my fps was shit in all but the smallest fights. When you finally see Retribution I don't think you will recognize sins except for the skybox. We'll see if I can keep the quality more consistent than the old project.
My small ships average 40k triangles. Polies haven't been relevant in major performance crunching in games for about a decade now (the vertex buffer was more or less the issue in wc3, for example). Control panel based forced AA murders sins performance, though. Keep that in mind. So does windowed.
Particles in sins will be your biggest cpu concern after pathing-related nonsense. I discovered this through rather extensive testing between Salvation's beginning and present day. I run a lot of tests. I'm a testing kind of guy. The one conclusion I made in regards to Salvation's particles when I began Retribution was that they were trash. Not just how they looked, but how they performed. Little did I realize just how bad they were.
Drag, size oscillator - stay away from these unless mandatory. Drag is kind of funky but it's also a cpu drag and I guarantee an animated texture will result in a superior effect. Size oscillator is something you should be a bit careful with. It's not bugged, like drag, but I used it extensively for the Undead in Salvation and it is in large part responsible for the UD slowdowns you see in Salvation. Krdax informed me that some modifiers actually leak memory. The only modifiers you generally need are Fade and Inflate, but color oscillator is handy too. All fine tuned animation should be done through crossblending and texture animations. I try to keep my actual systems very simple. Remember, all CPU activity is competing with each other, and particles are generally going to be CPU limited in sins.
Explosions are tough. I'm still working on mine. They're shaping up, but I don't like showing unfinished content publicly, so I won't. Take my word for it - Salvation's explosions sucked, Ret's are better but still got a ways to go. Those sparks in Salvation's explosions are particle murder. Drop the cluster texture, go for individuals (use the random sheet already in sins), and spawn only a handful. I don't let my explosions go over 60-70 particles peak and try to stick around 30 concurrent. Both peak and concurrent matter, of course. As a reference, a vanilla capital is something like 150 and one of the Bailknight capital is something absurd like 300. Managing these numbers isn't just about performance, it's about keeping particles from vanishing.
Trails. I use fake trails now. I absolutely despised those unavoidable gaps with my missile trails. Particle trails don't update fast enough for my long-ranged high-speed projectiles. Spamming particles and elongating them just doesn't work for me. So what I do is I have trails attached to the emitter but with reverse speeds, and live very shortly. All of the Anahn missiles ditched trails entirely, I just give them missile bodies with engine graphics now.
Bloom and additive blend very readily. Therefore, plan ahead and avoid spawning a lot of particles at once. Avoid noisy textures so they don't mesh with each other (or take advantage of that behavior).
My huge muzzle flashes for the Anahn rely on a set of animated textures, a flash, and sometimes a secondary flame or a set of bailknight-esque TEC sparks. For my average Anahn gun I try to stick to 4-6 particles peak for a muzzle flash. My most expensive muzzle flashes using a very nice fire sheet I ripped from Tera peak at around 15-20 for the most absolute expensive ones. My old Salvation muzzles often used up to 30-40 because I couldn't get a good fire stream with static textures without spawning a lot of them. They looked ugly as shit regardless. Absolutely do not spawn sparks, much less sparks with drag, on rapid fire guns like Bailknights does. They look a little silly, and they all add up with a lot of ships firing at once. If you need sparks, make a animated sheet. I've got a few I can share if you want but they aren't perfect for this purpose.
The most consistently expensive particle system in Salvation was, according to the Dev exe, the shells. Yes, the shells with only 2 particles. The thing was, with ships averaging 100 guns, firing in bursts of up to 12 every few seconds per gun, those shells added up. This is why my projectiles all move extremely fast (and not just because they fire slugs at 3/4 the speed of light in canon) - to reduce how many weapons are up at once. Minimum speed of 11000 or so for Anahn bullets. For the Xy and Undead, this is less necessary, because they have less guns. Thus, I can ration more particles to them. Old UD plasma torpedo - 200 particle uptime. New one - 15. Mostly because I gutted the trail and gave it a fake one and ditched the sparks.

No animated textures there, just a shockwave-esque texture with reverse inflation, a shitty looking flare I'll probably scrap or change, and the fake trail. The bloom handles my glow for such a small projectile.
The library of textures I've built up comes from dozens of games and many of them are either hand made or significantly edited to suit my purposes. I'm not an artist, but I do know how to make some custom fireballs in 3ds max. I just don't have the tools currently. I might make some of them public when the project is done, but I'd like to keep a few little secrets a surprise for the big show should it get finished.

^ Afterburn and FumeFX are the primary 3ds max plugins used to make animated sheets. Somewhere I think a commercial cd or set of cds exist of such texture sheets, but my belgian grips aren't limitless and I have yet to locate such a valuable resource. I did, however, use these to create sheets for Brood War. Here's a gif of one of them.

Significant problem, however - in my experience sins has a texture frame limit of around 20. Around this point the particles tend to flip out and roll over or just break and make strange squares. So, a good chunk of my resources don't work, and to make a high-detail set like that gif above would be a total waste. Therefore, a good particle system in sins would make use of several such texture sheets. The end result of a sheet looks a lot like this.

Keep in mind, that image above has some frame clipping. Don't be an intern. Fade those out if it happens - they look horrible.
Well, I don't like making big posts too often. My developer series will cover a good chunk of my testing, my analysis, my particle creation, and a whole lot of swearing, when it is public. Until then, I did post a small particle-related video in my Retribution thread if you want to learn more.
For the record, my environment (and conditions for my performance) are rather different from standard sins. I can't make any guarantees that employing a huge revision like what I do would have a similar return in vanilla. I can only say that it made an enormous difference in my own project under very heavily scrutinized tests.