As both are database function 1st take average function then apply round. In this case round is applying before average so you have to create custom column.
Use the below query in your custom column to do average of particular column.
In my case, “Monthly_Profit” is my table name and “Profit” is column name. save the query, then click more option on “Profit” column, then got to advanced function and select the function “Round” in numeric data type, then drag AVERAGE column into the function and specify the number to round. In my case am rounding to 2, save and generate the report to see the result.