Recommended jetting for 500AE's as supplied by Paul Rooney is
38mm carb: Slide 2.5 - 3.0, pilot 50 - 55, needle jet R2, needle 6DH3, main 400, float level 17mm - 19mm
40mm carb: Slide 2.5, pilot 55- 65, needle jet AA0, needle 7DH3, main 370 - 380, float level 19mm
Try changing the pilot jet first. My experience with going to Boyensens is on my 82 430XC is I had to go to a smaller pilot and main to get the bike to run well at small throttle opening. My 500 AE runs Boyensens and behaves well on the 38mm carb setting listed above at sea level.
Good luck