Weekly Development Notes #28 – We’re messing around with the cache again

Covering the week of August 3rd – 7th, 2020.

Hello and thank you for checking out our 28th development update of the FLIP Fluids liquid simulation addon for Blender! The Blender Market Summer Sale took place last week and development mainly consisted of handling Q&A for sales questions and making some updates to the FLIP Fluids simulation cache structure. We’re having a bit of an elevated level of support requests this week, but that is expected after a sale as we have many new users getting acquainted with the addon.

Updating the simulation cache

Last week I have been continuing work on updating how the simulation cache is structured. Specifically the part of the cache that stores settings and mesh data exported from the Blender scene. This is a part of the code that is running in the background when you see the Exporting animated data… message when starting a bake.

This is a large component of the FLIP Fluids addon code that needs to be rewritten and updated in order to be able to store different types of data to support more force field modes. And while I’m working on this area of the addon, it’s a good time to make some design decisions to accommodate a few other feature requests and quality-of-life requests.

This part of the addon has been rewritten before to fix issues. In the first release of the addon, a poor design decision in how the cache was structured could cause a complex scene with animated meshes to freeze Blender for minutes before the simulation began, or even cause the system to run out of memory (very annoying). It was a lot of work, but it was great to get that fixed! But now we’re seeing even more complex scenes that are causing issues during export and we once again need to redesign the system.

Why does the addon need to export scene and mesh data?

When we first started working on the addon, one of the most important design decisions and workflow features we wanted was the ability to have Blender completely usable while the simulator was running. This meant we needed to have the simulator run in a background thread, but due to the way that the Blender Python API for addons works, there are some complications with implementation. The Blender Python API is not thread-safe, which means we cannot access Blender’s data from the background simulation thread. Due to not being able to access Blender from the simulation thread, we needed to export scene data into our own file structure and format so that it can be sent to the simulator.

One way to get around this could have been to stop the thread after each frame and use the main Blender thread to gather next frame’s settings, and then send that to the simulation thread so that it can continue running. A problem with this approach is that exporting the data for a frame can cause Blender to stutter for short periods of time after each simulation frame. It also would cause the timeline frame to jump to new frames while you are working on your scene. Very annoying!

In order to have Blender completely usable without stuttering while the simulator is running in the background, we decided to export scene data for the entire simulation frame range ahead of time before the simulation starts running. The exported data includes:

  • All settings and properties for the domain and simulation objects
  • Keyframed settings (or settings with f-curve animation)
  • Static mesh data (meshes that don’t move)
  • Keyframed mesh data (or meshes that are transformed with f-curve animation)
  • Animated mesh data (for meshes with animation more complex than keyframes/f-curves)

After exporting this data, the simulator has everything it needs to run the entire simulation from beginning to end! All of this data is exported to files that are located in the cache, specifically to the cache_directory/export folder. Some of this data is exported as JSON text files, some as binary OBJ files (.bobj), and some as our own custom data file formats.

Why do we need to update the scene and mesh exporter?

To add more complex features, we need to start adding support for exporting more types of data than the kinds we listed in the previous section. For example, to support curve guided force fields we need to export curve data that is structured a bit differently than the triangle mesh data exported for regular mesh objects. As we add more export data types, the cache becomes more complex and is more difficult to manage code-wise under our current export system. Some of the new types of data we want to be able to export for the simulator are:

  • Curve Data – For curve guided force fields.
  • Local Axis Data – To store the local orientation of a mesh. This can be used to add a feature to emit an inflow in the direction of a local X/Y/Z axis, which is a requested feature. We already have a feature to emit in the direction of a target object, but it would be good workflow to also have the option to simply emit from an axis. This is also needed for force fields that rely on orientation.
  • Centroid Data – To store a single point as the centre of the object. This is to optimize how target objects work in the addon. The most optimal object to use for a target is an Empty, which is just a single point, but adding a centroid data type would make using any other object (such as a mesh) just as optimal.
  • Vertex Data – To store just the vertices of an object. This can also be used to store vertex attributes such as speed/colour which can be used for interesting effects. This format can also be used to efficiently export particle system data.
    • A requested feature is to be able to set a mesh or particle system and have the simulator emit droplets randomly from the points for creating rain effects.
  • Settings controlled by drivers or the NLA editor – At the moment, the addon only supports exporting animated settings that are keyframed (or contain f-curves). We’re seeing some use cases where users want to control options using drivers, or edited in the NLA panel, or even options that are scripted. For example, a use case could be animating an inflow Enabled/Disabled option that is triggered by the audio of a sound file.

Those are just a few more data types in addition to the existing triangle meshes and settings that we already export. But this gets complicated to manage under the current system due to the fact we need three different formats for each type depending on whether the object is static, keyframed, or if it contains complex animation.

What changes are we making to the scene cache export?

Under our current export system, the exported scene and mesh data is stored on the filesystem in individual files. For static meshes, just a single binary OBJ (.bobj) is stored. For keyframed meshes, a .bobj file and a file containing a list of transform matrices for each frame is stored. For a complex animated mesh, a separate .bobj file is stored for each frame.

As we add more data types, this means storing many more small files, which can be slow and very inefficient to write. For example, in a recent FLIP Fluids tutorial (Make SEXY sweets with the FLIP Fluids addon for Blender), part of this required exporting 150 separate keyframed objects (sprinkles) to the cache. At two files for each keyframed object, that’s a total of 300 small files that need to be written, which can be a bit slow. If those 150 objects were instead animated without keyframes, and there were 250 frames to export that would mean 250 * 150 = 37500 separate files! That’s way too inefficient.

We’re make some changes to how the export cache is stored on the filesystem, which will improve write performance and also make the structure more organized and easier to develop. Instead of writing many small files, we’re transitioning to store the export data into a single database file using Python’s built-in SQLite module.

It’s been years since I’ve touched a database or have written an SQL query, so I’ve been doing a bit of brushing up on databases during last week and this week.

Will these changes break older simulation caches? At the moment, it looks like we can still easily support caches that were created with older FLIP Fluids versions in the next version. These changes only affect the exported scene data (which can just be re-exported to a database under the new version) and not the simulation output data.

Next FLIP Fluids Experimental Build

No significant changes or bug fixes have been committed to to the force field branch over the last week, so there won’t be an updated build for this week. Most of the week’s development has been put towards the cache update, which is not yet ready to be merged into the experimental branch for testing. Maybe this large change will be ready for next Wednesday or the week after!

New FLIP Fluids Complete Guide Video: Mastering FOAM

The new FLIP Fluids Complete Guide video on FOAM is now online! This is the first of a two part episode to teach you all about how to use our FLIP Fluids addon whitewater feature. Enjoy watching!

The next part will cover the whitewater bubble, spray, and dust particles! For more learning resources, check out this page: https://github.com/rlguy/Blender-FLIP-Fluids/wiki/Video-Learning-Series

5 4 votes
Article Rating