Working with the Sydney Infrastructure team and Arx, to indentify scalability limitations in the current infrastructure, it was stated that our current Logging infrastructure couldn't handle another several hundred to 1000 active tenants, nevermind another 4000-5000 tenants that WCA is talking about adding to WCH.

This can be partially accomplished by asking squads to revisit the amoung of log/trace information they are producing, and avoid too verbose / unnecessary log output

Notes from Sydney-Infra (Lee/Blake/Di)

Current logging can barely keep up (eg, with LN at peak usage)

Requesting multiple ES clusters from platform team would be costly, consider alternate index architecture on the fly, will index as you request

P2 will provide/manage kafka cluster for WCH (shared kafka cluster) along with dedicated graylog instance from platform team

Sydney Infra to come up with proposed options, to get us to the scale (thousands of tenants) that we need, for WCA integration rollout.