When you want to insert or update a shape column using the FileGDB API, you must use one of the ShapeBuffer helper classes when you call the Row::SetGeometry function. You should not use the ByteArray class.
We ship sample programs which illustrate the use of the several of the ShapeBuffer derived classes. There is also extensive documentation of the shape buffer format included with the API. It is critically important that you understand the shape buffer format. This is essentially the same format as the shape buffers used in shape files, but with some Geodatabase extensions (e.g., curves, multipatches, etc.).
The API does not provide a way to directly use or convert other shape buffer formats to that which is required by the API.