Face recognition and people search
zeitkapsl supports face recognition and search. Privacy-preserving on-device machine learning models in the iOS, Android and Desktop apps analyze your photos for faces without the photos ever leaving your device unencrypted. The recognized faces are then stored alongside the photo metadata encrypted and are synced across your devices. There's a technical deep-dive into how we implemented privacy-preserving face recognition in our blog.
In short, zeitkapsl automatically finds and groups similar looking faces into persons. Since the face recognition is not perfect, sometimes, multiple such "persons" are created for the same person. You can manually merge them together in the mobile apps as well as the web apps.
Where do I see the detected persons?
In the mobile apps, navigate to the search category view (🔍 in the bottom right), then tap "People". In the web-app, navigate to the "People" view through the navigation menu or open https://app.zeitkapsl.eu/person. If persons have been detected, you will see a list of persons here.
Why are no persons detected?
The person detection requires multiple occurrences of the same face to be able to correctly group them together. The larger your library of faces, the better the detection will be. Add more photos of the persons you want to detect to your library. If you still do not see any persons, you can try to manually trigger face recognition.
How do I trigger face recognition?
On Android and iOS, the face recognition pipeline is run automatically overnight when your phone is charging and idle, together with the other search indexing. So in normal use, you don't have to do anything, just wait for the indexing to complete. On the desktop app, you can manually trigger the indexing, including the face recognition, by clicking the "Index" button in the upload view. You can also manually trigger indexing in the mobile apps by navigating to the search category view (🔍 in the bottom right) and clicking the ✨ button in the top right.
Why does zeitkapsl not find all pictures of a person? Why are not all faces on an image detected?
Machine-learning based face recognition is very sensitive to poor quality images or partly covered faces. Since a few bad face recognitions can break the automatic person grouping, the apps are filtering face recognitions based on the following criteria:
- Size: Even if an image is high-resolution, if a face is too small, it is excluded from the face recognition. If the image is stored locally, we try to use the original image for face recognition where possible. Otherwise, to minimize data usage, we use the same high-resolution preview image that is used for the gallery. Therefore, we recommend running the indexing on the device where your photos are stored. For example, when importing a bunch of photos through the desktop app, we recommend triggering face recognition in the desktop app after the images are imported. This way, the best possible image is used for face recognition.
- Noise: If an image is too noisy in the area of a face, the face is excluded from the face recognition. This can especially happen for images taken with a smartphone in low light conditions.
- Under-/Over-exposure: When the area around a face is too dark or too bright, the face is excluded from the face recognition.
- Angle: In our testing, the face recognition model we are using proved to be unreliable for faces facing sideways. Therefore, we are excluding faces where the angle between the face and the camera is too large.
- Aspect ratio: A very narrow bounding box around a face indicates the face being partly covered or taken at a very steep angle. Therefore, faces with a very large aspect ratio are excluded from the face recognition.
- Detection confidence: Our face detection model returns a confidence score for each detected face. If the confidence score is too low, the face is excluded from the face recognition, since it often indicates that the detected face is either non-human (e.g. animals) or a false positive. On heavily retouched images, detection confidence can in rare cases be low, even though the face is human.
There are multiple people entries shown for the same person
Since the detection and grouping are not perfect, we have adjusted the grouping algorithm to be conservative in order to avoid grouping people who are not the same person. This means that sometimes the same person is shown multiple times. You can manually merge these entries:
Merging people entries
- navigate to the people page.
- Select the person you want to merge the other person entries into. In the Android app, long press the face. In the iOS App, long press the face and select "merge". In the web app, navigate your cursor over the person card and click the select-checkbox in the top right.
- Select the other person entries belonging to the same person by tapping/clicking the faces.
- Click the merge button (top-right in the web app, bottom right in the iOS app, menu in the top right in the Android app).
- Confirm the merge.
Renaming people
Navigate to the people page. In the web app, click on the person you want to rename. You will see a view with all pictures of that person. Click on the name of the person and enter the new name, confirm with enter.
In the mobile apps, long press the face and select "rename" in the menu. Enter the new name and confirm.
An incorrect face has been assigned to a person.
You can correct this in the web app:
- Open the picture containing the incorrectly assigned face.
- Open the info-view by clicking on the i-symbol.
- Find the incorrectly assigned person tag by hovering the tags with the mouse cursor until the incorrectly assigned face is highlighted.
- Click on the small x on the tag to remove the assignment.
- Optionally, you can reassign the tag by following the steps below
Manually assign faces to people.
Currently, manually assigning faces in possible in the web app.
- Open the picture containing a face that is not yet showing up in the people search.
- Open the info-view by clicking on the i-symbol.
- You will see a "not assigned" tag for each of the detected, but not yet assigned faces. Hover your cursor over the tag to identify the correct tag and click it.
- Select the person you want to assign the face to or create a new person by clicking the "new person" button.