Hello.
In case of big amount of data I would suggest don’t use API – this is not it should be used for. However, there are three alternatives:
1) If you are doing initial import (i.e. you need to import 7m records only once) I’d propose to write plain Doctrine fixture to load all entities – expected speed is about 20k-200k entities/hour (depends on complexity of your entity). Advantage of this approach – it’s very simple and relatively fast, disadvantage – it’s solution to use only once.
2) If you are planning to do such import on a regular basis, but with smaller amount of entities (via UI or CLI) – I’d suggest to use built in import/export feature. Here is related webinar and here is documentation about import acceleration – expected speed is about 10k-100k entities/hour (depends on complexity of your entity). Advantage of this approach – it’s flexible, extendable and reusable, disadvantage – a bit slower speed.
3) If you are want to do that import very fast, or you need to load millions of entities on a daily basis – I’d suggest to write custom CLI command for this import and use plain SQL (without Doctrine layer), expected speed is about 1m-10m entities/hour. Advantage of this approach is speed, disadvantage – you have to manually run additional operations after the import to maintain DB consistency, e.g. do search reindex, update data audit tables and do other stuff usually handled by Doctrine layer.
And, of course, you can use any combination of these approaches – e.g. do initial import via plain SQL and then perform daily updated via import/export.
Dima Soroka 7 years, 11 months ago.
