Pricing examples
Example 1
If you are operating a small Conveyor environment where you have 1 environment and 2 Airflow DAGs running with 5 tasks each. All these tasks use an mx.medium instance that has 1 core and they run for 5 hours each day.
This result in 5 *30 = 600 core hours used per month for batch applications. I you have one Spark Streaming application running on an mx.medium driver and 2 mx.medium executors you use 3 (mx.medium core hours) *24 (hours in a day) * 30 days = 2160.
Datafy license costs
In total you will have used 2760 instance hours this month, you fall into the free category.
AWS Cloud cost
This AWS cloud cost is an estimation and may vary from the real cost. An mx.medium job is hosted on a class of instance which m6i.mxlarge is a part of this instance costs about $0.214 in the Ireland AWS region. An mx.medium job uses a fourth of this instance so we can estimate the cost of that instance size at 0.0535. For 2760 instance hours this results in a cost of 147.66 when running on-demand. If we run on spot we can reduce this cost up to 90%, but let’s take a conservative 70% reduction: 44.298.
Total cost
Our total cost would be 147.66 when running on-demand, and 44.298 when running on spot instances.
Example 2
You are operating a medium Conveyer environment, where you have a development, staging and production environment. Each of these runs 10 Dags with an average of 20 tasks.
Half of these jobs run spark with one mx.small driver and 5 mx.medium executors for 30 min.
The other halve are container jobs running mx.medium instance for 10 min.
Spark executor: 10 dags * 10 tasks * 5 executors * 1/2hour* 30days = 7500 mx.medium hours
Spark driver: 10 * 10 * 1/2 hours *30 = 1500 mx.small hours = 750 mx.medium hours.
Container jobs: 10 * 10 * 1/2 hours *30 = 1500 mx.medium hours
Monthly you will use 9750 mx.medium hours of which 5000 are free. You will pay 4750 *0.05 = 237.5 in licensing cost.
Example 3
You are operating a medium Conveyor environment where you are deploying some streaming applications. Your streaming pipeline contains 5 steps where each steap uses an mx.small driver and 4 mx.medium executors.
Spark executor costs: 5 * 4 executors * 24hour* 30days = 14400 mx.medium hours
Spark driver: 5 * 24 *30 = 3600 mx.smallhours = 1800 mx.medium hours.
Total: 16240 hours, of which 5000 are free you will pay 11240 hours * 0.05 = 562 in licensing cost.