{"id":9083,"date":"2024-06-21T17:49:42","date_gmt":"2024-06-21T15:49:42","guid":{"rendered":"https:\/\/eurocc.nscc.sk\/?p=9083"},"modified":"2024-06-21T18:21:06","modified_gmt":"2024-06-21T16:21:06","slug":"implementacia-metody-ciastocne-riadeneho-ucenia-uni-match-do-metody-frame-field-learning-pre-ulohu-extrakcie-budov-z-leteckych-snimok","status":"publish","type":"post","link":"https:\/\/eurocc.nscc.sk\/en\/implementacia-metody-ciastocne-riadeneho-ucenia-uni-match-do-metody-frame-field-learning-pre-ulohu-extrakcie-budov-z-leteckych-snimok\/","title":{"rendered":"Semi-Supervised Learning in Aerial Imagery: Implementing Uni-Match with Frame Field learning for Building Extraction<\/strong>"},"content":{"rendered":"
\n
\n
\n
\n

Semi-Supervised Learning in Aerial Imagery: Implementing Uni-Match with Frame Field learning for Building Extraction<\/strong><\/p>\n\n\n\n

<\/p>\n<\/div><\/div>\n\n\n\n

Building extraction in GIS (geographic information system) is pivotal for urban planning, environmental studies, and infrastructure management, allowing for accurate mapping of structures, including the detection of illegal constructions for regulatory compliance. Integrating extracted building data with other geospatial layers enhances the understanding of urban dynamics and spatial relationships. Given the scale and complexity of these tasks, there is a growing need to automate building extraction using deep learning techniques, which offer improved accuracy and efficiency in handling large-scale geospatial data.<\/p>\n\n\n\n

<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/a>
illustrative image<\/figcaption><\/figure>\n<\/div>\n<\/div>\n\n\n\n

State-of-the-art image segmentation models primarily output in raster format, whereas GIS applications often require vector polygons. One such method to meet this requirement is Frame Field learning, which addresses the gap between raster format outputs of image segmentation models and the vector format needed in GIS. This approach significantly enhances the accuracy of building vectorization by aligning with ground truth contours and provide topologically clean vector objects.<\/p>\n\n\n\n

These models are trained using a 'supervised learning' method, necessitating a large amount of labeled examples for training. However, obtaining such a significant volume of data can be extremely challenging and expensive. A potential solution to this problem is 'semi-supervised learning,' a method that reduces reliance on labeled data. In semi-supervised learning, the model is trained with a mix of a small set of labeled data and a larger set of unlabeled data. Hence, the goal of this collaboration between the Slovak National Competence Center for High-Performance Computing and Geodeticca Vision s.r.o. was to identify, implement, and evaluate an appropriate semi-supervised method for Frame Field learning.<\/p>\n\n\n\n

The aim of this cooperation between the National Competence Center for HPC and Geodeticca Vision s.r.o. was to identify, implement and evaluate a suitable partial tutor learning method for Frame Field learning.<\/p>\n\n\n\n

Methods<\/h5>\n\n\n\n
Frame Field learning<\/h5>\n\n\n\n

The key idea of the frame field learning [1] is to help the polygonization method in solving ambiguous cases caused by discrete probability maps (output from image segmentation models). This is accomplished by introducing an additional output to the neural network of image segmentation, namely a frame field (see. Fig. 1), which represents the structural features and geometrical characteristics of the building.<\/p>\n\n\n\n

Frame fields<\/p>\n\n\n\n

Frame field is a 4-PolyVector field that assigns four vectors to each point on a plane. Specifically, the first two vectors are constrained to be opposite to the other two, meaning each point is assigned a set of vectors {u, \u2212u, v, \u2212v}. This approach is particularly necessary for buildings, as they are regular structures with sharp corners, and capturing directionality at these sharp corners requires two directions.<\/p>\n\n\n

\n
\"\"<\/a>

Figure 1: Visualization of the frame field output on the image from training set [1].<\/em><\/figcaption><\/figure><\/div>\n\n\n

Frame Field learning<\/p>\n\n\n

\n
\"\"<\/a>
Figure 2: Diagram of the frame field learning [1]<\/figcaption><\/figure><\/div>\n\n\n

The learning process of frame fields can be summarized as follows:<\/p>\n\n\n\n

    \n
  1. The network's input is a 3\u00d7H\u00d7W RGB image.<\/li>\n\n\n\n
  2. To generate a feature map, any deep segmentation model could be used, such as U-Net, which is then processed to output detailed segmentation maps.<\/li>\n\n\n\n
  3. The training is supervised with ground truth rasterized polygons for interiors and edges, utilizing a mix of cross-entropy and Dice loss for accurate segmentation.<\/li>\n\n\n\n
  4. To train the frame field, three losses are used:
    1. Lalign<\/sub> enforces alignment of the frame field to the tangent direction.<\/li><\/ol>
      1. Lalign90<\/sub> prevents the frame field from collapsing to a line field.<\/li><\/ol>\n
          \n
        1. Lsmooth<\/sub> measures the smoothness of the frame field.<\/li>\n<\/ol>\n<\/li>\n\n\n\n
        2. Additional losses, regularization losses, are introduced to maintain output consistency, aligning the spatial gradients of the predicted maps with the frame field.<\/li>\n<\/ol>\n\n\n\n

          Vectorization<\/p>\n\n\n

          \n
          \"\"<\/a>
          Figure 3: Visualization of the vectorization process [1]<\/figcaption><\/figure><\/div>\n\n\n

          The vectorization process transforms classified raster images into vector polygons using a polygonization method using the Active Skeleton Model (ASM). The principle of this algorithm is the iterative shifting of the vertices of the skeleton graph to their ideal positions. This method optimizes a skeleton graph - a network of pixels outlining the building's structure - created by a thinning method applied on a building wall probability map. The iterative shifting is controlled by a gradient optimization method aimed at minimizing an energy function, which includes specific components related to the structure and geometry being analyzed:<\/p>\n\n\n\n

            \n
          1. Eprobability \u2013<\/sub> fits the skeleton paths to the contour of the building interior probability map at a certain probability threshold, e.g. 0.5<\/li>\n\n\n\n
          2. Eframe field align <\/sub>aligns each edge of the skeleton graph to the frame field.<\/li>\n\n\n\n
          3. Elength<\/sub> ensures that the node distribution along paths remains homogeneous as well as tight.<\/li>\n<\/ol>\n\n\n\n

            UniMatch semi-supervised learning<\/strong><\/p>\n\n\n\n

            UniMatch [2], an advanced semi-supervised learning method in the consistency regularization category, builds upon the foundational principles established by FixMatch [3], a baseline method in this domain. primarily operates on the principle of pseudo-labeling combined with consistency regularization.<\/p>\n\n\n\n

            The basic principle of the FixMatch method involves generating pseudo-labels for unlabeled data from the predictions of a neural network. Specifically, for a weakly perturbed unlabeled input xw<\/sup><\/em> , a prediction pw<\/sup><\/em>pw is generated, which serves as a pseudo-label for the prediction of xwith<\/sup><\/em>, a strongly perturbed input. Subsequently, the loss function value, for example, cross-entropypw,  <\/sup>pwith<\/sup><\/em>is calculated, considering only areas from pw<\/sup><\/em>pw with a probability value greater than a certain threshold, e.g., >0.95. <\/p>\n\n\n\n

            UniMatch builds upon and extends the FixMatch methodology, introducing two core enhancements:<\/p>\n\n\n\n

              \n
            1. UniPerb (Unified Perturbations for Images and Features) - This involves applying perturbations at the feature level. Practically, this means applying a dropout function to the output (i.e., the feature) from the encoder layer of the neural network, randomly ignore features, which then proceed to the decoder part of the network, generating pfp<\/sup><\/em>.<\/li>\n\n\n\n
            2. Instead of using one strong perturbation, two perturbations are utilized. xs1 <\/sup><\/em>from xs2<\/sup><\/em>.<\/li>\n<\/ol>\n\n\n
              \n
              \"\"<\/a>
              Figure 4: (a) The FixMatch baseline (b) used UniMatch method. The FP denotes feature pertubation, w and s means weak and strong pertubation, respectively [2].<\/figcaption><\/figure><\/div>\n\n\n

              Ultimately, there are three error functions: crossentropy(pw,  <\/sup>pfp<\/sup><\/em>), cross-entropy(pw,  <\/sup>ps1<\/sup><\/em>), cross-entropy(pw,  <\/sup>ps2<\/sup><\/em>These are then linearly combined with the supervised error function.<\/p>\n\n\n\n

              T\u00e1to met\u00f3da v s\u00fa\u010dasnosti patr\u00ed medzi state-of-the-art met\u00f3dy u\u010denia s \u010diasto\u010dn\u00fdm u\u010dite\u013eom. Hlavnou v\u00fdhodou tejto met\u00f3dy je jej jednoduchos\u0165 pri implement\u00e1ci\u00ed a nev\u00fdhodou je jej citlivos\u0165 na v\u00fdber vhodnej slabej a silnej perturb\u00e1cie.<\/p>\n\n\n\n

              Integrating UniMatch Semi-Supervised Learning with Frame Field Learning<\/strong><\/h4>\n\n\n\n

              Implementation Strategy for UniMatch in Frame Field Learning<\/strong><\/h4>\n\n\n\n

              To integrate UniMatch into our Frame Field learning framework, we first differentiated between weak and strong perturbations. For weak perturbations, we chose basic spatial transformations such as rotation, mirroring, and vertical\/horizontal flips. These are well-suited for aerial imagery and straightforward to implement.<\/p>\n\n\n\n

              For strong perturbations, we opted for photometric transformations. These include adjustments in hue, color, and brightness, providing a more significant alteration to the images compared to spatial transformations. <\/p>\n\n\n\n

              Incorporating feature perturbation loss was a crucial step. We implemented this by introducing a dropout mechanism between the encoder and decoder parts of the network. This dropout selectively omits features at the feature level, which is essential for the UniMatch approach.<\/p>\n\n\n\n

              Regarding the dual-stream perturbations of UniMatch, we adapted our model to handle two types of strong perturbations. The dual-stream approach involves using the weak perturbation prediction as a pseudo-label and training the model using the strong perturbation predictions as loss functions. We have two strong perturbations, hence the term 'dual-stream'. Each of these perturbations contributes to the overall robustness and effectiveness of the model in semi-supervised learning scenarios, especially in the context of building extraction from complex aerial imagery.<\/p>\n\n\n\n

              Prostredn\u00edctvom t\u00fdchto \u00faprav bola UniMatch met\u00f3da \u00faspe\u0161ne integrovan\u00e1 do Frame Field learning algoritmu, \u010d\u00edm sa zv\u00fd\u0161ila jeho schopnos\u0165 efekt\u00edvne sprac\u00fava\u0165 a u\u010di\u0165 sa z anotovan\u00fdch a hlavne neanotovan\u00fdch d\u00e1t.<\/p>\n\n\n\n

              Experiments
              Dataset
              Labeled Data<\/strong><\/h4>\n\n\n\n

              Our labeled data comes from three different sources, which we'll detail in the accompanying Table 1.<\/p>\n\n\n

              \n
              \"\"<\/a>
              Table 1: Overview of 3 data sources of labeled data used for training the models with details.<\/figcaption><\/figure><\/div>\n\n\n

              Unlabeled Data<\/strong><\/h4>\n\n\n\n

              For the unlabeled dataset, we selected high-quality aerial images from Geodetick\u00fd a kartografick\u00fd \u00fastav (GK\u00da) [6], available for free public use. We specifically targeted a diverse area of 7000 km2<\/sup>ensuring a wide representation of various landscapes and urban settings.<\/p>\n\n\n\n

              Data Processing: Patching<\/strong><\/h4>\n\n\n\n

              We processed both labeled and unlabeled images into patches of size 320x320 px. This patch size is specifically chosen to match the input requirements of our neural network. From the labeled data, this process resulted in approximately 55,000 patches. Similarly, from the unlabeled dataset, we obtained around 244,000 patches.<\/p>\n\n\n\n

              Training setup
              Model Architecture<\/strong><\/h4>\n\n\n\n

              We designed our model using a U-Net architecture with an EfficientNet-B4 backbone. This combination provides a good balance of accuracy and efficiency, crucial for handling the complexity of our segmentation tasks. The EfficientNet-B4 backbone was specifically chosen for its optimal balance between memory usage and performance. In Frame Field learning, U-Net architecture has been shown to be highly effective, as evidenced by its strong performance in prior studies.<\/p>\n\n\n\n

              Training Process<\/strong><\/h4>\n\n\n\n

              For training, we used the AdamW optimizer, which combines the advantages of Adam optimization with weight decay, aiding in better model generalization. To prevent overfitting, we implemented L2 regularization. Additionally, we used the ReduceLROnPlateau learning rate scheduler. This scheduler adjusts the learning rate based on validation loss, ensuring efficient training progress.<\/p>\n\n\n\n

              Semi-Supervised Learning Adjustments<\/strong><\/h4>\n\n\n\n

              A key aspect of our training was adjusting the ratio of unlabeled to labeled patches. We experimented with ratios ranging from 1:1 to 1:5 (labeled:unlabeled). This variability allowed us to explore the impact of different amounts of unlabeled data on the learning process. It enabled us to identify the optimal balance for training our model, ensuring effective learning while leveraging the advantages of semi-supervised learning in handling large and diverse datasets.<\/p>\n\n\n\n

              Model evaluation<\/strong><\/h4>\n\n\n\n

              In our evaluation of the building footprint extraction model, we chose metrics that precisely measure how well our predictions align with real-world structures.<\/p>\n\n\n\n

              Intersection over Union (IoU)<\/strong><\/h4>\n\n\n\n

              K\u013e\u00fa\u010dovou metrikou, ktor\u00fa sme vyu\u017e\u00edvali je metrika s n\u00e1zvom Intersection over Union (IoU). Po\u010d\u00edta zhodu medzi predikciami modelu a skuto\u010dn\u00fdm tvarom budov. Hodnota sk\u00f3re IoU bl\u00edzka 1 znamen\u00e1, \u017ee na\u0161e predikcie s\u00fa podobn\u00e9 skuto\u010dn\u00fdm budov\u00e1m. T\u00e1to metrika je nevyhnutn\u00e1 na pos\u00fadenie geometrickej presnosti pre segmentovan\u00e9 oblasti, preto\u017ee odr\u00e1\u017ea presnos\u0165 vyt\u00fd\u010denia hran\u00edc budov. Okrem toho, vyhodnoten\u00edm pomeru spr\u00e1vne predikovanej oblasti ku kombinovanej oblasti (zjednotenie oblasti predikcie a skuto\u010dnej oblasti), n\u00e1m IoU poskytuje jasn\u00fa mieru efektivity modelu v zachyt\u00e1van\u00ed skuto\u010dn\u00e9ho kontextu a tvaru budov v komplexnej mestskej krajine.<\/p>\n\n\n\n

              Precision, Recall and F1<\/strong><\/h4>\n\n\n\n

              Precision measures the accuracy of the model's building predictions, indicating the proportion of correctly identified buildings out of all identified buildings, thereby reflecting the model's specificity. Recall assesses the model's ability to capture all actual buildings, with a high recall score highlighting its sensitivity in detecting buildings. The F1 Score combines precision and recall into a single metric, offering a balanced view of the model's performance by ensuring that high scores result from both high precision and high recall.<\/p>\n\n\n\n

              Complexity Aware IoU (cIoU)<\/strong><\/h4>\n\n\n\n

              We also utilized Complexity Aware IoU (cIoU) [7]. This metric addresses a shortfall in IoU by balancing segmentation accuracy and the complexity of the polygon shapes. While IoU alone can lead models to create overly complex polygons, cIoU ensures that the complexity of the polygons (number of vertices) is kept realistic, reflecting the typically less complex structure of real buildings.<\/p>\n\n\n\n

              N Ratio Metric<\/strong><\/h4>\n\n\n\n

              The N ratio metric was an additional component of our evaluation strategy. It contrasts the number of vertices in our predicted shapes with those in the actual buildings [7]. This helps in understanding whether our model accurately replicates the detailed structure of the buildings.<\/p>\n\n\n\n

              Max Tangent Angle Error<\/strong><\/h4>\n\n\n\n

              To ensure clean geometry in building extraction tasks, accurately measuring contour regularity is essential. The Max Tangent Angle Error (MTAE) [1] metric is designed to address this need by supplementing the Intersection over Union (IoU) metric. It specifically targets the limitation of IoU, where segmentations with rounded corners may receive higher scores than those with more precise, sharp corners. By evaluating the alignment of edges through the comparison of tangent angles at sampled points along predicted and ground truth contours, MTAE effectively penalizes inaccuracies in edge orientation. This focus on edge precision is critical for producing clean vector representations of buildings, emphasizing the importance of accurate edge delineation in segmentation tasks.<\/p>\n\n\n\n

              Evaluation Process<\/strong><\/h4>\n\n\n\n

              Natr\u00e9novan\u00e9 modely boli testovan\u00e9 na ve\u013ekej d\u00e1tovej mno\u017ene leteck\u00fdch sn\u00edmok v plnej ve\u013ekosti (namiesto mal\u00fdch \u010dast\u00ed, pomocou ktor\u00fdch bola sie\u0165 tr\u00e9novan\u00e1). Tak\u00e9to testovanie poskytuje presnej\u0161ie zobrazenie re\u00e1lnych pou\u017eit\u00ed tak\u00fdchto modelov. Na extrakciu budov zo sn\u00edmok v plnej ve\u013ekosti sme pou\u017eili techniku posuvn\u00e9ho okna, \u010d\u00edm boli vytvoren\u00e9 predikcie po jednotliv\u00fdch segmentoch obr\u00e1zku. Na okraje prekr\u00fdvaj\u00facich sa segmentov bola pou\u017eit\u00e1 pokro\u010dil\u00e1 priemerovacia technika, d\u00f4le\u017eit\u00e1 pre minimaliz\u00e1ciu ne\u017eiad\u00facich efektov a zachovanie konzistentnosti v r\u00e1mci predik\u010dnej mapy. V\u00fdstupn\u00e1 predik\u010dn\u00e1 mapa v plnej ve\u013ekosti bola n\u00e1sledne vektorizovan\u00e1 do presn\u00fdch vektorov\u00fdch polyg\u00f3nov s pou\u017eit\u00edm algoritmu Active Skeleton Model (ASM).<\/p>\n\n\n\n

              Results<\/strong><\/p>\n\n\n

              \n
              \"\"<\/a>
              Tabu\u013eka 2: V\u00fdsledky tr\u00e9novania modelov pre z\u00e1kladn\u00fd pr\u00edstup (u\u010denie s u\u010dite\u013eom) a pr\u00edstupy u\u010denia s \u010diasto\u010dn\u00fdm u\u010dite\u013eom s r\u00f4znymi podielmi pou\u017eit\u00fdch anotovan\u00fdch a neanotovan\u00fdch obr\u00e1zkov.<\/figcaption><\/figure><\/div>\n\n\n

              The results from our experiments, reflecting performance of segmentation model trained under different conditions, reveal significant insights (see Table 2). We evaluated the model's performance in a baseline scenario without semi-supervised learning and in scenarios where semi-supervised learning was applied with varying ratios of labeled to unlabeled data (1:1, 1:3, and 1:5).<\/p>\n\n\n\n

                \n
              1. IoU: <\/strong>Starting from the baseline IoU of 80.50%, we observed a steady increase in this metric as we introduced more unlabeled data into the training process, reaching up to 85.77% with a 1:5 labeled to unlabeled ratio<\/li>\n\n\n\n
              2. 2.\tPrecision, Recall, and F1 Score: <\/strong>The precision of the model, which measures how accurate the predictions are, improved from 85.75% in the baseline to 90.04% in the 1:5 ratio setup. Similarly, recall, which indicates how well the model can find all relevant instances, slightly increased from 94.27% to 94.76%. The F1 Score, which balances precision and recall, also saw an improvement from 89.81% to 92.34%. These improvements suggest that the model became more accurate and reliable in its predictions when semi-supervised learning was used.<\/li>\n\n\n\n
              3. N Ratio a cIoU: <\/strong>The results show a notable decrease in the N Ratio from 2.33 in the baseline to 1.65 in the semi-supervised 1:5 ratio setup, indicating that the semi-supervised model generates simpler, yet accurate, vector shapes that more closely resemble the actual structures. This simplification likely contributes to the enhanced usability of the output in practical GIS applications. Concurrently, the complexity-aware IoU (cIoU) significantly improved from 48.89% in the baseline to 64.75% in the 1:5 ratio, suggesting that the semi-supervised learning approach not only improves the overlap between the predicted and actual building footprints but also produces simpler vector shapes, which are closer to real-world buildings in terms of geometry.<\/li>\n\n\n\n
              4. Mean Max Tangent Angle Error<\/strong> MTAE: <\/strong>The Mean MTAE's reduction from 18.60\u00b0 in the baseline to 17.45\u00b0 in the 1:5 semi-supervised setting signifies an improvement in the geometric precision of the model's predictions. This suggests that the semi-supervised learning model is better at capturing the architectural features of buildings with more accurately defined angles, contributing to the production of topologically simpler and cleaner vector polygons.<\/li>\n<\/ol>\n\n\n\n

                Training on High-Performance Computing (HPC) Machine<\/strong><\/h4>\n\n\n\n

                HPC Configuration<\/strong><\/h4>\n\n\n\n

                Our training was conducted on a High-Performance Computing (HPC) machine equipped with substantial computational resources. The HPC had 8 nodes, each outfitted with 4 NVIDIA A100 GPUs with 40GB of VRAM, 64 CPU cores, and 256GB of RAM. For task scheduling, the system utilized Slurm.<\/p>\n\n\n\n

                PyTorch Lightning Framework<\/strong><\/h4>\n\n\n\n

                We employed the PyTorch Lightning framework, which offers user-friendly multi-GPU settings. This framework allows the specification of the number of GPUs per node, the total number of nodes, various distributed strategies, and the option for mixed-precision training.<\/p>\n\n\n\n

                Experiences with Slurm and PyTorch Lightning<\/strong><\/h4>\n\n\n\n

                When training on a single GPU, our Slurm configuration was as follows:
                #SBATCH –partition=ngpu
                #SBATCH –gres=gpu:1
                #SBATCH –cpus-per-task=16
                #SBATCH \u2013mem=64000<\/p>\n\n\n\n

                In PyTorch Lightning, we set the trainer as: Trainer<\/em>:<\/p>\n\n\n\n

                trainer = Trainer(accelerator=”gpu”, devices=1)<\/p>\n\n\n\n

                Since, here, we allocated one GPU from four available in one node, we allocated 16 CPUs from 64 available. Therefore, for the data loaders, we assigned 16 workers. Since semi-supervised learning uses two data loaders (one for labeled and one for unlabeled data), we allocated 8 workers to each. It was critical to ensure that the total number of cores for the data loaders did not exceed the available CPUs to prevent training crashes.<\/p>\n\n\n\n

                Distributed Data Parallel (DDP) Strategy<\/strong><\/h4>\n\n\n\n

                Using PyTorch Lightning's Distributed Data Parallel (DDP) option, we ensured each GPU across the nodes operated independently:<\/p>\n\n\n\n