Place the halved baby potatoes in a large pot and cover with cold water. Add a pinch of salt and bring to a boil. Cook until the potatoes are tender when pierced with a fork, about 12 to 15 minutes.
Drain the potatoes in a colander and let them cool for 10 minutes in a large mixing bowl.
In a small bowl, whisk together the olive oil, lemon zest, lemon juice, capers, and minced shallot. Season with salt and pepper.
Pour the dressing over the slightly warm potatoes and toss gently to coat.
Add the chopped parsley and dill, then toss again until the herbs are evenly distributed. Taste and adjust seasoning if needed.