To answer your last question first, the 64-bit Background Geoprocessing applies only to geoprocessing tools being executed in the background. It can be used whether the tools are executed directly or in ModelBuilder or python. ArcGIS for Desktop is a 32-bit application but when you install the patch and execute the tools in the background, they run in 64-bit.
When you run the model, is the progress dialog opening? If so, then the model is still running in the foreground. How are you executing the model? If you are executing it from the ModelBuilder dialog, I think it always runs in the foreground so you need to execute it as you would any geoprocessing tool. If the model is running in the foreground, then do the following:
1. Right-click on the model and open the "Properties" dialog.
2. Under the "General" tab, uncheck the setting to "Always run in the foreground" and apply the change.
3. Now execute the model by double-clicking on it (not by opening the ModelBuilder window).
If the model is already running in the background and you are still getting "out of memory" errors, then you have two main choices:
1. Process the data in smaller subsets (this should be straightforward since you already have the model); or
2. Try to get Vince's suggested approach to work.