Accepting ordinary food after the fact isn’t idolatry by itself. Knowingly joining the ritual could be.
In the Bible, idol worship means participating in worship, praying to another god or offering a sacrifice as an act of allegiance. Being near an object or food connected to another religion doesn’t count.
Paul deals with this in 1 Corinthians 10:20-28. Food bought or served can be eaten without interrogating where it came from, but he draws the line at joining a sacrificial meal as worship.
If it’s still unclear, ask what accepting the food means to them. Don’t go against your conscience, but don’t panic either.